As the development of 3D film technique, the editing method of binocular video image has increasingly been the focus in recent years. However, it is relatively difficult to extend the existing methods of editing 2D video image to binocular video image because additional information such as parallax needs to be processed.
The integration of binocular stereo video scenes refers to that: two segments of binocular stereo video A and B are given to integrate the moving object of video A into the dynamic scene of video B with maintaining the consistency in color, parallax and time to yield an integration result of binocular stereo video scenes.
There have been many publications in the fields of computer vision and computer graphics concerning the method of editing 2D video image and the method of editing binocular video image. So we will not try to list exhaustively all of them here. Nevertheless, the works related to the video editing with time consistency will be introduced here as the focus and the integration methods of video image scene and the methods of editing binocular image will also be involved.
The ordinary method of editing monocular video has been studied to a relatively great deal in recent years. In 2002, Chuang et al. applied the bidirectional optical flow in an algorithm of inter-frame propagation of tri-map. In 2009, Bai et al. utilized a set of local classifiers and an optical flow algorithm to successfully achieve a result of better time consistency. In 2010, Bai et al. further applied the color flow in the propagation step of tri-map for accurate extracting video object. These methods can achieve a relatively good effect compared to ordinary editing of monocular video. However, these mask propagation methods can't be applied directly in the method of editing binocular stereo video. In 2008, Sand et al. proposed a practical method of long-term tracking. In 2009, Lee et al. utilized the flow field of scale-invariant feature transform (that is, SIFT) to put forward a video stabilization algorithm. And Bhat et al. applied the optical flow algorithm in an algorithm of editing multiple gradient-domain video in 2010. In 2012, Chen et al. applied the concept of maintaining local linear structure in editing and propagation of video. These algorithms of editing video with time consistency mainly focus on object tracking and content editing. But the purpose of the present invention is to simultaneously edit the shape and color of a binocular stereo video object.
The method of integrating video image scene is a conventional one of editing video image. In 2003, Perez et al. put forward a method of Poisson integration which solves a Poisson equation to integrate seamlessly the region into a new scene. In 2009, Jia et al. utilized accurate object extraction to further improve successfully the effect of Poisson integration. In 2009, Farbman et al. utilized the median coordinates to greatly increase successfully the rate of integration. In 2010, Xie et al. extended the median coordinate integration to the fusion of monocular video. In 2013, Chen et al. utilized the optical flow to successfully extend the median coordinate integration to video compositing. However, these methods haven't realized to maintain the time consistency of the extracted moving objects.
The method of editing binocular video image also relates to the works of the present invention. In 2010, Lo et al. proposed a method name as “billboard” to compose the object and scene of binocular image. In 2012, Luo et al. developed an iterative method of integrating binocular video scene which can adjust deformation and color simultaneously. In 2012, Niu et al. put forward a method of binocular image deformation according to user interaction and disparity map. All these methods are focusing on the editing operation of binocular image. Wang et al. in 2008 and Weng et al. in 2006 separately developed a method of image deformation. In 2010, Lang et al. put forward a method of video retargeting which utilized an optical flow to successfully ensure the inter-frame deformation and propagation.